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1.  Objectives 

The  main  objective  of  the  research  program  has  been  to  develop  and  implement  a 
three-dimensional,  block-structured  (zonal)  solution  scheme  for  Euler 
equations  on  computers  with  multiple  processors.  The  methodology  is  aimed  at 
developing  computational  techniques  for  solving  three-dimensional,  transonic 
flow  problems  around  complex  geometries .  For  solving  problems  such  as 
transonic  flow  around  an  aircraft,  one  expects  to  employ  many  computers  for 
the  solution  of  a  single  problem.  The  developed  scheme  involves  the 
partitioning  of  a  large  aerodynamics  problem  into  several  smaller  problems 
where  each  sub-problem  is  defined  for  a  particular  flow  region.  Each  of  these 
problems  are  then  solved  on  individual  processors.  The  boundary  condition 
between  the  neighboring  flow  regions  are  communicated  between  these  processes. 
This  work  has  been  the  continuation  of  a  development  over  the  years  for 
generating  grids  for  complex  geometries  and  solving  large  problems  for  such 
geometries  by  using  a  zonal  approach  and  finally  studying  the  appropriate 
computer  architectures  for  solving  such  problems.  The  main  publications 
describing  the  research  work  are: 

1.  Ecer,  A.,  Spyropoulos,  J.T.,  and  Maul,  J.D.,  "A  Three-Dimensional, 
Block-Structured  finite  Element  Grid  Generation  Scheme,"  AIAA 
Journal,  Vol.  23,  Oct.  1985,  pp.  1483-1490. 

2.  Ecer,  A.  and  Spyropoulos,  J.T.,  "B1 ock-Structured  Solution  Scheme  for 
Analyzing  Three-Dimensional  Transonic  Potential  Flows,"  AIAA  Journal, 
Vol.  25,  Oct.  1987,  pp.  1292-1300. 

3.  Ecer,  A.,  Spyropoulos,  J.T.  and  Rubek,  V.,  "Block-Structured  Solution 
of  Euler  Equations  for  Transonic  Flows,"  AIAA  Journal,  Vol.  25,  Dec. 
1987,  pp.  1570-1576. 
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4.  Ecer,  A.,  Spyropoulus,  J.T.,  Sims,  M.,  Parallel  Processing  Techniques 
tor  the  solution  of  Euler  Equations,  AIAA  Paper  88-0620.  AIAA  26th 
Aerospace  Sciences  Meeting,  Reno,  Nevada,  January  11-14,  1988. 

2.  Description  of  the  Method  of  Solution 

The  method  of  solution  is  documented  in  several  papers  in  detail.  Here  a 
short  summary  of  the  methodology  is  provided: 

a)  61 ock-Structured  Grid  Generation  Scheme  (ref.  1). 

Flow  field  around  a  complex  geometry  is  represented  by  an  assembly  of  flow 

i 

regions.  Each  region  corresponds  to  a  simpler  geometry  (a  block)  which  can  be 
defined  easily.  These  blocks  are  attached  to  each  other  for  describing  the 
complex  flow  field.  The  block-structure  can  be  irregular  i.e.  blocks  can  be 
attached  to  each  other  in  an  unstructured  fashion.  The  procedure  keeps  track 
of  the  block-structure  at  all  times.  In  designing  a  computational  grid,  one 
specifies  the  grid  required  for  each  block.  The  blocks  are  automatically 
connected  to  each  other.  Besides  representing  complex  geometries,  the  blocks 
are  used  to  model  specific  flow  structures.  For  example,  a  block  is  placed  to 
include  the  flow  region  where  a  shock  occurs.  One  can  then  specify  the 
required  grid  refinement  around  the  shock  and  this  grid  is  automatically 
connected  to  the  neighboring  flow  regions.  This  methodology  is  also  useful 
when  one  starts  with  a  computational  grid  and  modifies  it  locally  after 
observing  the  accuracy  of  the  obtained  results.  Any  local  grid  modifications 
is  propagated  to  the  neighboring  blocks  automati cal ly . 

b)  Block-Structure  Solution  Schemes  (ref  2,3). 


After  the  computational  grid  is  generated  in  terms  of  a  series  of  blocks,  the 
solution  of  Euler  equation  is  defined  in  terms  of  the  solution  of  the 
equations  for  each  of  the  blocks  with  interface  boundary  conditions  between 


the  neighboring  blocks.  The  steady-Euler  equations  are  solved  by  using  a 
relaxation  scheme  where  the  velocity  vector  is  decomposed  by  using  a  Clebsch 
transformation  as  follows: 


ij  =  Vd  +  Svp 

where  S  is  the  enhopy,  d  and  n  are  Lagrange  multipliers.  The  Euler  equations 
are  written  in  terms  of  these  variables  as  follows: 
v(pv<j>)  =  -  v(pSVn) 
v.  lPuS)  -  0 

i 

pu  .  V13  =  -  p/R 

t 

where  p  is  the  density,  p  is  the  pressure  and  R  is  the  gas  constant.  For 
blocks  with  cut  rctationbity  (Constant  entropy)  one  needs  to  solve  only  the 
first  equation.  The  boundary  conditions  between  the  neighboring  blocks 
requires  the  continuity  of  S  and  n •  For  potential  equation,  only  the 
continuity  of  tp  is  required.  The  solution  scheme  require  iterative  solution 
of  these  equations  in  each  of  the  blocks  together  with  the  iteration  of  the 
boundary  conditions  between  the  neighboring  blocks.  An  efficient  methodology 
for  iterating  the  boundary  conditions  for  $,  S  and  n  have  been  developed, 
c)  Parallel  Processing  for  Block-Structured  Solution  of  Euler 
Equations  (ref. 4): 

The  developed  methodology  is  well  suited  for  parallel  processing.  For  a  given 
computer  with  many  processors,  the  blocks  are  distributed  among  the 
processors.  The  information  exchange  between  the  neighboring  processors 
involve  only  the  exchange  of  boundary  conditions  which  is  a  relatively  small 
set  of  data.  In  terms  of  parallel  processing,  two  types  of  computers  were 
tested.  For  a  computer  with  distributed  memory  (Intel-IPSC)  each  of  the 
blocks  is  stored  on  an  individual  computer  with  its  own  memory.  For  a 
computer  with  shared  memory  (IBM  3090),  the  memory  is  shared  by  several 
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processors.  In  the  first  case,  the  numbers  of  processors  and  the  blocks  are 
matched.  In  the  second  case,  the  number  of  blocks  is  much  larger  than  the 
number  of  processors. 

The  objective  of  this  part  of  the  study  was  to  investigate  the  applicability 
of  parallel  processing  for  solving  large  aerodynamics  problems.  The 
methodology  is  quite  general  and  numerical  results  have  been  obtained  for 
several  practical  problems  in  a  parallel  environment.  The  completed  work 
resents  one  of  the  few  applications  of  relaxation  schemes  in  a  parallel 
computing  environments 

3.  Summary  of  the  Research: 

The  research  work  on  the  block-structured  solution  of  Euler  equations  with 
parallel  processing  progressed  along  four  main  areas: 

A.  Implementation  of  the  B1 ock-Structured  Solution  Scheme  on  a  Intel 
hypercube  with  distributed  memory  and  sixteen  processors. 

B.  Implementation  of  the  Block-Structured  Solution  Scheme  on  a  IBM  3090 
computer  with  shared  memory  and  four  processors. 

C.  Configuration  of  Grid  Generation  Studies. 

A.  Implementation  of  the  Block-Structured  Solution  Scheme  on  a  Intel 
Hypercube . 

This  activity  involved  the  solution  of  aerodynamics  problems  on  a 
computer  with  many  processors  each  with  its  own  individual  memory  (distributed 
memory).  The  example  in  figure  1  shows  the  sixteen  block  model  of  a  wing  body 
problem.  In  this  case,  each  block  is  stored  on  an  separate  computer.  Grids 
are  generated  on  a  block-by-block  bases  and  the  solution  is  obtained  for  each 
block.  The  only  exchange  of  information  between  the  neighboring  blocks  is  the 
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FIG.  IF •  Node  configuration  on  INTEL  Computer  for  the 
sixteen  block  model 
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boundary  conditions.  This  application  is  one  of  the  first  implementations  of 
a  three-dimensional  CFD  code  on  a  Intel  Hypercube  computer. 

The  following  papers  were  written  describing  this  work: 

1.  Ecer,  A.,  Spyropoulus,  J.T.,  Sims,  M.,  Parallel  Processing  Techniques 
for  the  solution  of  Euler  Equations,  AIAA  Paper  88-0620.  AIAA  26th 
Aerospace  Sciences  Meeting,  Reno,  Nevada,  January  11-14,  1988. 

2.  Ecer,  A.,  Spyropoulus,  J.T.,  Block  Structured  Solution  of  Euler  and 
Nai ver-Stokes  Equations,  2nd  Int.  Symp.  on  Domain  Compositions 

t 

Methods,  USC^  Los  Angeles,  January  14-16,  1988.  (invited  paper) 

i 

3.  Ecer,  A.,  Spyropoulus,  J.T.,  and  Sims,  M.,  The  Solution  of  Euler 
Equations  on  a  Hypercube,  3rd  Conf.  on  Hypercube  Concurrent  Computers 
and  App.,  Caltech,  Pasadena,  January  19-20,  1988. 

4.  Ecer.  A.  and  Akay,  H.U.,  A  Block-Structured  Finite  Element  Solution 
of  Viscous  Internal  Flows,  Int.  Conf.  on  Comp.  Meth.  in  Flow 
Analysis,  September  4-8,  1988,  Okayoma,  Japan,  (invited  paper) 

5.  Ecer,  A.,  &  Spyropoulos,  J.T.,  "Parallel  Processing  Schemes  for  the 
Block-Structured  Solution  of  Transonic  Flows,"  Society  Computer 
Simulation  Summer  Conference,  Ontario,  Canada,  July  23-29,  1987. 


B.  Implementation  of  the  Block-Structured  Solution  Scheme  on  IBM-3090 
Computer . 

In  this  case,  the  same  problem  was  implemented  on  a  computer  where  four 
processors  share  the  same  large  memory  as  shown  in  figure  2.  Many  blocks  are 
solved  on  each  of  the  processors  where  one  can  address  several  gigabytes  of 
memory  in  an  efficient  manner  by  using  the  developed  scheme.  Presently  a  12 
processor  version  of  this  application  is  attempted.  We  have  been 
collaborating  with  IBM  research  center  in  Kingston  on  this  activity. 


(512  +  4096  Megabytes) 


Block-str”ctured  solution  scheme  on  an  IBM  3090-E  computer 
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The  impl ementation  of  the  scheme  on  parallel  processors  for  solving  large 
problems  was  investigated  by  using  the  developed  capability.  Numerical 
results  were  obtained  for  a  nacelle  and  a  wing-nacelle  configuration  which 
involve  over  150  blocks  as  shown  in  figure  3.  Presently  a  F-16  geometry 
aircraft  (over  500,000  grid  points)  is  under  investigation  as  shown  in  figure 
4. 

The  following  papers  was  prepared  describing  this  activity: 

1.  Chang,  S.M.,  Ecer,  A.  and  Spyropoulos,  J.T.,  "The  Use  of  IBM  Parallel 

FORTRAN  and  Multitasking  facility  with  a  Domain  Decomposition 
1 

Approach  for  the  Euler  Equations,"  IBM  Symposium  on  Parallel 
Processing  Poughkeepsie,  New  York,  Oct.  19-21,  1988. 

2.  Ecer,  A.,  Spyropoulos,  J.T.  &  Chang,  S.M.  "Parallel  Processing 
Schemes  for  the  Block-Structured  Solution  of  Transonic  Flows  II," 
Society  of  Computer  Simulation  Summer  Conference,  Seattle, 

Washington,  July  25-28,  1988. 

3.  Chang,  S.M.,  Ecer,  A.  and  Spyropoulos,  J.T.,  "Parallel  Processing 
Techniques  of  the  Euler  Equations  on  the  IBM  3090  VF  Computer,"  The 
Second  International  Conference  on  Vector  &  Parallel  Computing  Issues 
in  Applied  Research  and  Development,  Tromso,  Norway,  June  6-10,  1988. 

4.  Chang,  S.M.,  Ecer,  A.  and  Spyropoulos,  J.T.,  "Parallel  Processing 
Techniques  on  the  IBM  3090VF  Computer  System,"  7th  International 
Conference  on  Finite  Element  Methods  in  Flow  Problems,  Huntsville, 
Alabama,  Apr.  3-7,  1989. 

5.  Spyropoulos,  J.T.,  Ecer,  A.  Rout,  R.K.,  and  Badesha,  S.S.,  "Block- 
Structured  Solution  of  Three-Dimensional  Transonic  Flowfields  for 
Wing/Pylon/Nacelle  configurations,"  AIAA  Paper  89-2940,  25th  Joint 
Propulsion  Conference,  Monterey,  California,  July  10-14,  1989. 


Figure  Aa. 


Block-Structure  for  the  F-16  aircraft 


Figure  4b.  Block-Structured  Grid  for  the  F-16  Aircraft 
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C.  Grid  Generation  Studies. 

The  continuation  of  the  grid  generation  studies  were  basically  focused  on 
testing  the  accuracy  and  efficiency  of  the  scheme  for  complex  geometries. 

The  following  papers  were  written  describing  this  activity: 

1.  Ecer,  A.,  Spyropoulos,  J.T.,  Application  of  a  Three-Dimensional 
Finite  Element  Grid  Equation  Scheme  for  Analyzing  Transonic  Flow 
Around  an  Aircraft,  Conf.  on  Automated  mesh  Generation  and 
Adaptation,  Grenoble,  France,  Oct.  1987.  (invited) 

2.  Ecer,  A.,  Spyropoulos,  J.T.  and  Bulbul  E.,  Application  of  a  Three- 
Dimensional  Finite  Element  Grid  Generation  Scheme  for  F- 16  Aircraft 
Conf i gurati on ,  2nd  Int.  Conf.  on  Num  Grid.  Generation  in  Comp.  Fluid 
Dynamics,  Miami  Beach,  Dec.  5-8,  1988. 

3.  Ecer,  A.,  Akay,  H.U.  and  Spyropoulos,  J.T.,  "Appl i cations  of  a  Three- 
Dimensional  Finite  Element  Grid  Generation  Scheme  to  Flow  Problems," 
5th  International  Conference  on  Numerical  Methods  in  Laminar  and 
Turbulent  Flows,  Montreal,  Canada,  July  6-10,  1987. 

4.  Conclusions 

The  main  objective  of  the  research  has  been  the  development  of  methodology  for 
solving  large  transonic  flow  problems  with  complex  geometries.  One  can 
develop  a  basic  computational  procedure  for  solving  a  particular  flow  problem 
around  a  two-dimensional  simple  geometry  (eq.  an  airfoil).  However,  it  is  not 
always  easily  possible  to  perject  the  applicability  of  this  procedure  for 
solving,  complex,  three-dimensional  flow  problems.  We  have  been  using  finite 
element  methods  for  solving  CFD  problems  basically  due  to  their  flexibility 
in  modeling  complex  geometries  (unstructured  grids).  However,  our  experience 
with  large  problems  indicated  that  it  is  very  difficult  to  treat  large 
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problems  as  a  single  problem.  Even  if  one  can  generate  "a  grid"  for  a  complex 
geometry  it  requires  several  iterations  to  ensure  that  this  is  "an  acceptable 
grid"  in  terms  of  providing  disired  accuracy  with  available  computational 
resources.  In  terms  of  solving  Euler  equations,  we  have  also  reduced  the 
problem  to  the  solution  of  potential  or  non-i sentropi c  potential  equation  for 
the  majority  of  the  flow  field  with  the  solution  of  the  Euler  equations  only 
at  certain  critical  regions.  In  terms  of  accuracy  of  the  solutions  in 
isentropic  flow  regions,  the  solution  of  potential  equations  are  quite 
satisfactory  with  coayser  grids  then  the  ones  used  for  Euler  equations 
docummented  in  the  literature.  In  the  developed  block-structured  solution 
scheme,  one  can  define  which  specific  regions  (blocks)  will  be  analyzed  by 

using  the  Euler  equations  and  provide  desired  grid  accuracy  in  these  regions. 

The  block-structure  solution  schemes  become  even  more  powerful  when  the 
solution  of  Nai ver-Stokes  equations  are  sought  which  requires  considerably 
refined  grids. 

The  parallel  processing  of  the  Euler  equations  proved  to  be  a  natural 
extension  of  the  developed  technology.  While  the  computational  speed  and  size 
of  computers  is  expected  to  increase  in  coming  years,  one  can  easily  expect 

8  9 

grids  with  10  -  10  grid  points  to  be  still  quite  expensive  on  a  single 

processor.  There  is  agreement  among  many  researchers  that  parallel  processing 
may  be  the  most  feasible  way  for  solving  such  large  problems.  It  is  important 

to  note  that  if  one  is  going  to  solve  a  large  problem  on  parallel  computers 

one  should  consider  the  methodology  which  will  work  efficiently  on  such 
machines.  As  can  be  seen  from  the  present  research,  the  grid  generation,  the 
solution  scheme  and  the  computer  implementation  must  be  considered  as  a  single 
problem.  The  conversion  of  an  existing  computational  procedure  and  a  computer 
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code  to  a  parallel  machine  does  not  always  produce  impressive  results.  Some 
of  the  current  issues  in  CFD  such  as  adaptive  grids,  efficiency  and  accuracy 
of  computational  schemes  has  to  be  studied  in  conduction  their  with 
application  to  parallel  computers.  For  example,  many  computational  schemes 
which  have  been  developed  for  structured  grids  are  being  revised  in  terms  of 
their  application  to  unstructured  grids.  One  expects  a  similar  activity  in 
the  future,  in  terms  of  implementing  different  schemes  for  parallel  computers. 
Instead,  it  seems  more  logical  to  develop  computati onal  schemes  with  the 
objective  that  they  w>i  1 1  work  on  parallel  computers.  The  completed  work 
provides  such  an  example  where  a  complete  methodology  is  developed  for  solving 
CFD  problems,  in  particular  transonic  flows,  in  a  parallel  computing 
envi roment . 


